CP-13581 linux-pkg: skip rebuilding windows-connector when its cached artifact is already up to date - #407
Conversation
f31f0e7 to
d0a21e4
Compare
be78e4c to
f243840
Compare
f243840 to
719b072
Compare
719b072 to
7ce82a9
Compare
prakashsurya
left a comment
There was a problem hiding this comment.
I agree with seb's latest comment..
IMO, if we really want to keep the post-push build trigger on the app-gate git repository, then I think this is overly complicated.. and really, we want a way to denote the build as "skipped".. instead of the S3 file manipulation.. and do nothing when it's skipped, as the prior artifact will already be preserved "forever"..
With that said, since we want to tightly control when we do rebuilds of the connector, I might suggest an even more simple approach.. and disable the post-push trigger entirely, and instead rely on folks to manually trigger new builds.. then, we can carefully control when we rebuild it, and tie that into w/e "recertification" process we currently have (correct me if this doesn't make sense, I'm unfamiliar with the current certification process that's mentioned in the PR description).. and the build script can just be simple like it is today, and always do a build..
To me, this feels awfully complicated for what essentially equates to "do nothing, if connector version hasn't changed".. there's no need to copy the S3 artifacts around..
514711f to
01deca0
Compare
… artifact is already up to date PR URL: #407
01deca0 to
2965e34
Compare
… artifact is already up to date PR URL: #407
2965e34 to
baab544
Compare
… artifact is already up to date PR URL: #407
baab544 to
4647325
Compare
… artifact is already up to date PR URL: #407
4647325 to
b7229dd
Compare
… artifact is already up to date PR URL: #407
b7229dd to
46c827b
Compare
… artifact is already up to date PR URL: #407
46c827b to
c79ec9d
Compare
… artifact is already up to date PR URL: #407
c79ec9d to
194b461
Compare
… artifact is already up to date PR URL: #407
194b461 to
c3f58bb
Compare
… artifact is already up to date PR URL: #407
c3f58bb to
af2d7d0
Compare
… artifact is already up to date PR URL: #407
af2d7d0 to
09739b9
Compare
… artifact is already up to date PR URL: #407
09739b9 to
c99d270
Compare
Problem
The
windows-connectorpackage is rebuilt from scratch on everylinux-pkgpipeline run, even when its source hasn't changed since the last successful build. Unlike other linux-pkg packages, windows-connector's build toolchain (nsis/mono/mingw-w64) lives outside our own source and can drift independently of any windows-connector code change — rebuilding on every unrelated trigger risks silently shipping a different toolchain combination without anyone deciding to re-certify it. This also masks the underlying dependency-drift bug (see dlpx-app-gate#4648) — the failure only surfaces when a buildserver's live apt mirror happens to have moved out of sync with the previously-pinned Artifactory dependency zips.Solution
Add a new
project.ext.connectorDebVersioninappliance/host/windows/build.gradle, bumped manually whenever connector source changes (independent ofconnectorVersion, which is bumped once per release regardless of whether the connector itself changed). Changeconfig.sh'sbuild()so the package is only rebuilt when that version doesn't have a matching artifact yet, reusing existing linux-pkg infrastructure rather than adding anything new:latestpost-push artifact viaget_package_dependency_s3_url— the same lookup every linux-pkg package already uses to fetch its own build dependencies.windows-connector_<connectorDebVersion>_all.debalready exists there, skip the build entirely and leave a.build_skippedmarker file in the artifacts directory instead of trying to reuse/republish anything itself — devops-gate#4772 teaches the shared pipeline to recognize that marker, report a normal JenkinsNOT_BUILTresult, and (for the multi-package orchestrator) resolve the existing artifact directly for downstream consumers.gradlew build+createDebPackage) as before. No explicit upload is needed — the shared post-push Publish stage already syncs the result to S3 and updateslatestfor us, exactly like every other package, when the build wasn't skipped.latestpointer) is ever missing for any reason, it just falls through to a normal build and republishes, rather than aborting.This supersedes two earlier approaches tried in this PR: a bespoke
release-de-images/windows-connector-cache/S3 path, and (after that) writing aBUILD_STATUSfile for a custom Slack notification line. Both were abandoned in favor of reusing existinglatest-artifact infrastructure and Jenkins' own supportedSUCCESS/NOT_BUILTresult codes — no new S3 location, no new status plumbing.Testing Done
Personal Jenkins controller testing: ops-jenkins's shared
generate-jobsseed job embeds each job's pipeline script from a fixed branch at seed time rather than reading it fresh per build, so exercising in-flight devops-gate changes (like devops-gate#4772) required a personal developer Jenkins controller pinned to that PR's branch (JenkinsManager configure -j ops -g devops-gate=<branch>,linux-pkg). Two runs there cover both sides of this change:windows-connectorbuild: build-package/custom #2 —.build_skippedmarker correctly detected,Finished: NOT_BUILT.Notes to Reviewers
This PR depends on devops-gate#4772 to be safe to land — without it, a skipped build's
.build_skippedmarker is never checked, and the shared pipeline falls through toPublishwith nothing to publish. Related, separately-tracked follow-ups:git reviewpre-commit check (and/or CI check) in dlpx-app-gate requiringconnectorDebVersionto be bumped wheneverappliance/server/connectororappliance/host/windowssource changes — without it, a merged windows-connector fix that forgets the version bump would be silently served from cache instead of shipping.Tracked under epic CP-13579, alongside CP-13582 (app-gate).